Companies and consumers increasingly depend on computers to process, distribute, and play back high quality video content. Engineers use compression (also called source coding or source encoding) to modify the bit rate of digital video. Compression decreases the cost of storing and transmitting video information by converting the information into a lower bit rate form. Decompression (also called decoding) reconstructs a version of the original information from the compressed form. A “codec” is an encoder/decoder system.
Compression can be lossless, in which the quality of the video does not suffer, but decreases in bit rates are limited by the inherent amount of variability (sometimes called source entropy) of the input video data. Or, compression can be lossy, in which the quality of the video suffers, and the lost quality cannot be completely recovered, but achievable decreases in bit rate are more dramatic. Lossy compression is often used in conjunction with lossless compression—lossy compression establishes an approximation of information, and the lossless compression is applied to represent the approximation.
A basic goal of lossy compression is to provide good rate-distortion performance. So, for a particular bit rate, an encoder attempts to provide the highest quality of video. Or, for a particular level of quality/fidelity to the original video, an encoder attempts to provide the lowest bit rate encoded video. In practice, considerations such as encoding time, encoding complexity, encoding resources, decoding time, decoding complexity, decoding resources, overall delay, and/or smoothness in quality/bit rate changes also affect decisions made in codec design as well as decisions made during actual encoding.
In general, video compression techniques include “intra-picture” compression and “inter-picture” compression. Intra-picture compression techniques compress a picture with reference to information within the picture, and inter-picture compression techniques compress a picture with reference to a preceding and/or following picture (often called a reference or anchor picture) or pictures.
For intra-picture compression, for example, an encoder splits a picture into 8×8 blocks of samples, where a sample is a number that represents the intensity of brightness or the intensity of a color component for a small, elementary region of the picture, and the samples of the picture are organized as arrays or planes. The encoder applies a frequency transform to individual blocks. The frequency transform converts an 8×8 block of samples into an 8×8 block of transform coefficients. The encoder quantizes the transform coefficients, which may result in lossy compression. For lossless compression, the encoder entropy codes the quantized transform coefficients.
Inter-picture compression techniques often use motion estimation and motion compensation to reduce bit rate by exploiting temporal redundancy in a video sequence. Motion estimation is a process for estimating motion between pictures. For example, for an 8×8 block of samples or other unit of the current picture, the encoder attempts to find a match of the same size in a search area in another picture, the reference picture. Within the search area, the encoder compares the current unit to various candidates in order to find a candidate that is a good match. When the encoder finds an exact or “close enough” match, the encoder parameterizes the change in position between the current and candidate units as motion data (such as a motion vector (“MV”)). In general, motion compensation is a process of reconstructing pictures from reference picture(s) using motion data.
The example encoder also computes the sample-by-sample difference between the original current unit and its motion-compensated prediction to determine a residual (also called a prediction residual or error signal). The encoder then applies a frequency transform to the residual, resulting in transform coefficients. The encoder quantizes the transform coefficients and entropy codes the quantized transform coefficients.
If an intra-compressed picture or motion-predicted picture is used as a reference picture for subsequent motion compensation, the encoder reconstructs the picture. A decoder also reconstructs pictures during decoding, and it uses some of the reconstructed pictures as reference pictures in motion compensation. For example, for an 8×8 block of samples of an intra-compressed picture, an example decoder reconstructs a block of quantized transform coefficients. The example decoder and encoder perform inverse quantization and an inverse frequency transform to produce a reconstructed version of the original 8×8 block of samples.
As another example, the example decoder or encoder reconstructs an 8×8 block from a prediction residual for the block. The decoder decodes entropy-coded information representing the prediction residual. The decoder/encoder inverse quantizes and inverse frequency transforms the data, resulting in a reconstructed residual. In a separate motion compensation path, the decoder/encoder computes an 8×8 predicted block using motion vector information for displacement from a reference picture. The decoder/encoder then combines the predicted block with the reconstructed residual to form the reconstructed 8×8 block.
I. Video Codec Standards.
Over the last two decades, various video coding and decoding standards have been adopted, including the H.261, H.262 (MPEG-2) and H.263 series of standards and the MPEG-1 and MPEG-4 series of standards. More recently, the H.264 standard (sometimes referred to as AVC or JVT) and VC-1 standard have been adopted. For additional details, see representative versions of the respective standards.
Such a standard typically defines options for the syntax of an encoded video bit stream according to the standard, detailing the parameters that must be in the bit stream for a video sequence, picture, block, etc. when particular features are used in encoding and decoding. The standards also define how a decoder conforming to the standard should interpret the bit stream parameters—the bit stream semantics. In many cases, the standards provide details of the decoding operations the decoder should perform to achieve correct results. Often, however, the low-level implementation details of the operations are not specified, or the decoder is able to vary certain implementation details to improve performance, so long as the correct decoding results are still achieved.
During development of a standard, engineers may concurrently generate reference software, sometimes called verification model software or JM software, to demonstrate rate-distortion performance advantages of the various features of the standard. Typical reference software provides a “proof of concept” implementation that is not algorithmically optimized or optimized for a particular hardware platform. Moreover, typical reference software does not address multithreading implementation decisions, instead assuming a single threaded implementation for the sake of simplicity.
II. Acceleration of Video Decoding and Encoding
While some video decoding and encoding operations are relatively simple, others are computationally complex. For example, inverse frequency transforms, fractional sample interpolation operations for motion compensation, in-loop deblock filtering, post-processing filtering, color conversion, and video re-sizing can require extensive computation. This computational complexity can be problematic in various scenarios, such as decoding of high-quality, high-bit rate video (e.g., compressed high-definition video). In particular, decoding tasks according to more recent standards such as H.264 and VC-1 can be computationally intensive and consume significant memory resources.
Some decoders use video acceleration to offload selected computationally intensive operations to a graphics processor. For example, in some configurations, a computer system includes a primary central processing unit (“CPU”) as well as a graphics processing unit (“GPU”) or other hardware specially adapted for graphics processing. A decoder uses the primary CPU as a host to control overall decoding and uses the GPU to perform simple operations that collectively require extensive computation, accomplishing video acceleration.
In a typical software architecture for video acceleration during video decoding, a video decoder controls overall decoding and performs some decoding operations using a host CPU. The decoder signals control information (e.g., picture parameters, macroblock parameters) and other information to a device driver for a video accelerator (e.g., with GPU) across an acceleration interface.
The acceleration interface is exposed to the decoder as an application programming interface (“API”). The device driver associated with the video accelerator is exposed through a device driver interface (“DDI”). In an example interaction, the decoder fills a buffer with instructions and information then calls a method of an interface to alert the device driver through the operating system. The buffered instructions and information, opaque to the operating system, are passed to the device driver by reference, and video information is transferred to GPU memory if appropriate. While a particular implementation of the API and DDI may be tailored to a particular operating system or platform, in some cases, the API and/or DDI can be implemented for multiple different operating systems or platforms.
In some cases, the data structures and protocol used to parameterize acceleration information are conceptually separate from the mechanisms used to convey the information. In order to impose consistency in the format, organization and timing of the information passed between the decoder and device driver, an interface specification can define a protocol for instructions and information for decoding according to a particular video decoding standard or product. The decoder follows specified conventions when putting instructions and information in a buffer. The device driver retrieves the buffered instructions and information according to the specified conventions and performs decoding appropriate to the standard or product. An interface specification for a specific standard or product is adapted to the particular bit stream syntax and semantics of the standard/product.
A conventional hardware encoder allows one acceleration entry point and exit point. For example, a main software module can activate, at the entry point, a motion-estimation accelerator to provide a motion vector field at the exit point. Alternatively, the main software module can activate, at the entry point, a fully accelerated video encoder to encode one picture and provide bit streams at the exit point. Thus, the encoder allows for little control so that the input follows a single, predetermined path to the output.
Unfortunately, video encoding process can involve very complex operations with strong data dependencies. As a result, a predetermined encoder accelerator should not be expected to deliver very high compression quality.
Greater flexibility is needed in the encoder in order to optimize acceleration.